Regret-Matching Bounds Bounds for Regret-Matching Algorithms

We study a general class of learning algorithms, which we call regret-matching algorithms, along with a general framework for analyzing their performance in online (sequential) decision problems (ODPs). In each round of an ODP, an agent chooses a probabilistic action and receives a reward. The particular reward function that applies at any given round is not revealed until after the agent acts. Further, the reward function may change arbitrarily from round to round. Our analytical framework is based on a set Φ of transformations over the agent’s set of actions. We calculate a Φ-regret vector by comparing the reward obtained by an agent over some finite sequence of rounds to the reward that could have been obtained had the agent instead played each transformation φ ∈ Φ of its sequence of actions. Regret-matching algorithms select the agent’s next action based on the vector of Φ-regrets together with a link function f . In this paper, we derive bounds on the regret experienced by (Φ, f)-regret matching algorithms for polynomial and exponential link functions and arbitrary Φ. Whereas others have typically derived bounds on distribution regret, our focus is on bounding the expectation of action regret. A simplification of our framework, however, yields bounds on distribution regret equivalent to (and in some cases slightly stronger than) others that have appeared in the literature.